System And Method For Presenting Content With Time Based Metadata

ABSTRACT

Systems and methods are described for presenting content with time-based metadata. Methods may comprise transmitting the time-based metadata to a playback device. The time-based metadata may comprise a plurality of time stamps that are triggerable at different points of a timeline of content and one or more of the time stamps may be associated with corresponding metadata content. The playback device may trigger a sequence of time stamps of the time-based metadata and the triggered sequence of time stamps may be synchronized with the occurrence of one or more events in the timeline of the content.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/670,883 filed on Aug. 7, 2017, which is a continuation of U.S. patentapplication Ser. No. 13/523,829 filed on Jun. 14, 2012, which claimspriority to Provisional U.S. Patent Application No. 61/497,023 filed onJun. 14, 2011, the entirety of which are incorporated herein byreference.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A-1D illustrate systems for including time based metadatainformation as a form of supplemental content in connection with thepresentation of media content, according to one or more embodiments.

FIG. 2 illustrates a method for generating time based metadata for amedia content, according to embodiments.

FIG. 3 illustrates a method for providing media data that is packaged orintegrated with time based metadata, according to an embodiment.

FIG. 4 illustrates a method for providing time based metadata to a mediasource, for transmission to a media playback device with media content,under an embodiment.

FIG. 5 illustrates a method for providing time based metadata to aplayback device that receives the media content from a media source,according to another embodiment.

FIG. 6A through FIG. 6F illustrate examples of media contentpresentation that includes metadata content, according to someembodiments described herein.

FIG. 7A through FIG. 7F illustrate examples of media contentpresentations that include metadata content for playback and/or paireddevices, according to some embodiments described herein.

FIG. 8 through FIG. 9C illustrate various interface tools forfacilitating different metadata placement or selection operators ingenerating metadata for a media content, according to an embodiment.

FIG. 10A-10B illustrate examples of media content interfaces generatedusing metadata content for a paired device, according to embodimentsdescribed herein.

FIG. 11 illustrates a method for providing media data to be shared orpublished, according to an embodiment.

FIG. 12A-12E illustrate example interfaces for providing media data tobe shared or published, according to an embodiment.

FIG. 13 is a block diagram that illustrates a computer system upon whichembodiments described herein may be implemented.

DETAILED DESCRIPTION

Embodiments described herein include a service that enables devices toutilize various web services independent of the communication protocoland format utilized by the devices.

In particular, devices that are capable of media playback and/or webbrowsing typically use communication protocols such as XML, EBIF(set-top boxes), REST or JSON. For such devices, content publishers makeavailable various forms of content, often to supplement or augment theuser's playback and/or web browsing experience. For example, cinematicstudios often publish movie previews and bonus clips on websites, whichcan be downloaded and viewed by a device that uses an XML communicationformat (e.g. ANDROID device), EBIF (set-top box) or JSON. Underconventional approaches, the content publishers are required to addressthe various communication protocols and formats used by the variousdevices individually. In contrast, embodiments described herein includea service that is capable of interfacing a device using any of multiplepossible communication protocols with a web service that may or may nothandle the particular communication protocol in question.

Moreover, under embodiments described, the web service providers mayupdate functionality and content provided without having to addressvarious possible communication protocols that are in use. Accordingly,otherwise incompatible devices that normally would not communicate witheach other may be able to connect, communicate and/or share information.

As used herein, BD refers to Blu-ray Disc®, and BD specification refersto the various documents that define the behavior and requirements of BDplayers, software and related systems, and, in particular, “SystemDescription Blu-ray Disc Read-Only Format: Part 3. Audio Visual BasicSpecifications”. The BD specification includes a section pertaining tovirtual file system (“VFS”) updates.

In some embodiments, reference is made to playback devices that operateas a Blu-ray player. A Blu-ray player generally supports a virtual filesystem (VFS) that utilizes files located on an optical disc itself, aswell as files that reside in a local storage area of the playbackdevice. The Blu-ray environment generally permits updates to theenvironment by way of new files and content. With reference toembodiments described herein, a Blu-ray player is able to retrieveinformation and data from a network, under programmatic instructionsthat originate from logic or software operating on the player, as wellas those executing with the disc. In such implementations, the Blu-rayplayer may be configured to access and retrieve metadata forpresentation in connection with media content that is provided at leastin part from the optical disc. In this way, the original contents andfunctionality of the optical disc can be augmented through the additionof content, namely, audio and video files, application files, or datafiles, such as those downloaded from a network server. Information thatupdates the playback environment of the player is received through theVFS of the playback device. With further reference to the Blu-rayenvironment, the service may perform functions, such as dynamicallygenerating a manifest for the data communicated to the player, as wellas a digital signature that is sufficiently in compliance with Blu-raySpecifications (e.g. See System Description Blu-ray Disc Read-OnlyFormat: Part Audio Visual Basic Specifications”). In a Blu-ray platformthe manifest defines modifications that are to be made to a file systemin the playback device. More specifically, the manifest file maps fileslocated on the Blu-ray disc and in local storage to their relativepositions and names in the virtual file system.

One or more embodiments described herein provide that methods,techniques and actions performed by a computing device are performedprogrammatically, or as a computer-implemented method. Programmaticallymeans through the use of code, or computer-executable instructions. Aprogrammatically performed step may or may not be automatic.

One or more embodiments described herein may be implemented usingprogrammatic modules or components. A programmatic module or componentmay include a program, a subroutine, a portion of a program, or asoftware component or a hardware component capable of performing one ormore stated tasks or functions. As used herein, a module or componentcan exist on a hardware component independently of other modules orcomponents. Alternatively, a module or component can be a shared elementor process of other modules, programs or machines.

Furthermore, one or more embodiments described herein may be implementedthrough the use of instructions that are executable by one or moreprocessors. These instructions may be carried on a computer-readablemedium. Machines shown or described with figures below provide examplesof processing resources and computer-readable mediums on whichinstructions for implementing embodiments of the invention can becarried and/or executed. In particular, the numerous machines shown withembodiments of the invention include processor(s) and various forms ofmemory for holding data and instructions. Examples of computer-readablemediums include permanent memory storage devices, such as hard drives onpersonal computers or servers. Other examples of computer storagemediums include portable storage units, such as CD or DVD units, flashmemory (such as carried on many cell phones and personal digitalassistants (PDAs)), and magnetic memory. Computers, terminals, networkenabled devices (e.g. mobile devices such as cell phones) are allexamples of machines and devices that utilize processors, memory, andinstructions stored on computer-readable mediums. Additionally,embodiments may be implemented in the form of computer-programs, or acomputer usable carrier medium capable of carrying such a program.

System Description

FIGS. 1A-1D illustrate a system for including time based metadatainformation as a form of supplemental content in connection with thepresentation of media content, according to one or more embodiments.Each of the embodiments described with respect to FIG. 1A-FIG. 1D,including components and programmatic processes described with eachembodiment, may be used individually or in combination with one another.In particular, embodiments described herein enable the rendering ofcontent, such as movies and/or television programs, to be enhanced withthe display of relevant metadata information that pertains to eventsthat are part of the content being watched. For example, the appearanceof an actor in the movie or program may be supplemented with relevantmetadata information that includes the name of the actor who is playingthe character, as well as additional information about the actor orcharacter. Likewise, (i) the presence of objects that appear in thecontent may be highlighted by metadata, particularly as to commercialproducts; (ii) the use of locations in the content may also besupplemented with information about such location; or (iii) music and/ora soundtrack playing in the audio background or as the primary audiotrack may be supplemented with metadata. Numerous other examples areprovided herein for enhancing or supplementing the presentation ofcontent, such as provided by movies and programs, with metadatainformation regarding the events that occur as part of the content beingwatched.

As used herein, metadata, in the context of movies and programs, refersto information and content that is displayed to the user, but which isnot part of the content itself. Rather, such metadata is delivered tothe user watching the movie or program as an additional or independentlayer.

In FIG. 1A, system 100 is described in the context of multipleoperational environments. According to embodiments, a system 100 may beoperational in one or more of the operational environments described.

System 100 includes metadata generation subsystem 110 and time metadataservice (or services) 120. The metadata generation subsystem 110processes media files 102 corresponding to movies and/or programs. Theoutput of the metadata generation 110 includes a metadata file 111,which includes a combination of time stamps 113 and associatedinformation 115. The metadata file 111 includes the time stamps 113 bywhich the sync time triggers 131 can be programmatically orautomatically activated by a playback device in order to cause therendering of associated information or content items for a particulartrigger.

Within the structure of the timeline for the media file, the time stamps113 are timed to coincide with the occurrence of a particular event inthe media content. Each time stamp 113 may signify the beginning of anevent, and last for a duration that coincides with the ending of theevent. For example, the introduction of an actor into a scene maysignify the beginning of a particular time stamp 113, and the actor'sexit from the scene may signify the end of the event associated with thetime stamp 113. Similarly, a time stamp 113 may be associated when musicor a particular song is played in the background of the media content,and a time stamp 113 may have the duration equivalent to the length ofthe song (e.g., the duration of time the music is playing). One frame orseries of frames in the media content can be associated with multipletime stamps 113. Furthermore, each time stamp 113 is associated with acorresponding item of information or content 115. Such information orcontent 115 can be displayed over the movie/program content on aplayback device, or on a separate, paired device, as a mechanism forproviding the viewer with additional information.

In some variations, the information and content displayed to the usercan be interactive. For example, the user may select a link presentedwith the information or content item 115 as a mechanism for viewingadditional information, or partaking in a supplemental activity (e.g.make online purchase).

In order to generate the metadata file 111, one or more embodiments mayinclude a combination of manual and programmatic resources comprisingthe metadata generation system 110. In one embodiment, aprogrammatic/manual enhancement 112 component may be used to facilitateoperators to identify time stamps 113, and associated information orcontent 115. With reference to an embodiment of FIG. 8, the programmaticmanual enhancement 112 may include a user interface tool 800 thatdisplays a general timeline 812 for a particular movie or program (e.g.the “Incredibles”). In an embodiment depicted, the user interface tool800 enables an operator to create events for scenes of the mediacontent. The events may correspond to an actor, playing a role in thecontent, appearing in a scene of the media content.

The user interface tool 800 may also include one or more event columns814. For example, the event columns 814 may include a list of actors, aswell as other events relating to the media content.

As also shown in FIG. 8, the timeline 812 extends horizontally relativeto the event columns 814. Within the timeline 812, various time stamps813 are identified by the operator each of which corresponds to one ormore events in the event column. Each time stamp 813 is represented bybrackets that can vary in dimension to represent a duration of the timestamp 813 over the course of the timeline 812. In one implementation, anoperator is enabled to (i) view a given scene or frame; (ii) view theexact time within the timeline in which the scene takes place; and (iii)have the ability to add new events, as well as to associate a particularsenior time stamp 813 within an existing event in the event column 814.

According to some embodiments, the metadata generation subsystem 110 canalso include programmatic resources to automate the generation of someor all of the metadata. In particular, programmatic automation 114 maybe used to identify some or all of the time stamps 113, as well as togenerate some or all of the information or content items associated withindividual time stamps 113. In one implementation, the programmaticautomation 114 uses a script for the media content in order to determinewhen events occur in the media content (e.g. the introduction of acharacter into a scene), what the event pertains to (e.g. the characterinvolved), and what the associated information or content item 115should include (e.g. the name of actor who plays the character of theevent). As an alternative or addition to the script, closed-captionedtext associated with the media content can be similarly used.

Still further, some embodiments may use image analysis to identifypersons or objects appearing in the content. For example, individualframes of the media content may be analyzed for faces, which are thencompared to a facial recognition database of actors, in order todetermine who the actor is that appears in the content.

As additional resources, the interface tool 800 can include informationsuch as the cast and the soundtrack list. When an operator creates atime stamp 813, the operator can create a record that contains orreferences information about the particular event. The information orcontent items can be listed for the operator to edit (see list ofrecords 818).

The time based metadata service 120 may utilize the metadata file 111 inone or more operational environments. In particular, the service 120 mayprovide (i) media content that is packaged with time based metadata (asgenerated by the metadata generation subsystem 110); or (ii) time basedmetadata in connection with media that is to be played back or renderedfrom a media source independent of the service 120.

In one operational environment, the service 120 combines the time baseddata 113 and media content 121 for broadcast and/or distribution. Anintermediate service or step 122 may retransmit or deliver the combinedtime based data 113 and media content 123 to a playback device 118. Forexample, a combined transmission time based metadata and media content121 may be supplied to a cable head end, which in turn broadcasts (orretransmits) the combined time based data 113 and media content 123 toits user population. As another example, the time based data and mediacontent 121 may be stored onto media (e.g. Blu-ray disc) and distributedthrough Blu-ray disc distribution channels. Other forms of media mayalso be used to store the combined time based data and media content121. For example, the combination of time based data and media content121 may be supplied as a download from a website. Still further, theplayback device 118 may correspond to a set-top box, and theintermediate service or step 122 may correspond to a cable head end thatdelivers the combined time based metadata and media 123 to the set-topbox in the appropriate format (e.g. EBIF format).

In another operational environment, the service 120 operates to supplythe time based metadata 125 independent of the media content thatpertains to the metadata. Accordingly, for a particular media content,the service 120 supplies a delivery node or trigger 126 with the timebased metadata 125 and media content identifier. The delivery node 126may include a broadcast or distribution mechanism 127, and anapplication or resource 129 to integrate or otherwise combine themetadata 125 with the underlying media content. In particular, themetadata 125 may be structured to provide time stamps 113 andinformation or content items 115 all along a timeline that is based onthe events that occur during the playback or rendering of the associatedmedia content. The integration resource 129 may synchronize the metadata125 with the media content in retransmitting, or otherwise makingavailable (e.g. download), in enhanced form, the media content 130 thatincludes metadata.

In this way, the delivery node or trigger 126 is able to deliver to aplayback device 128, a combination of integrated content or enhancedcontent 130 that includes the time based metadata and the associatedcontent.

Various mediums of playback may be encompassed by an operational modesuch as described with delivery node 126. For example, the delivery node126 may correspond to a cable head end, and the playback device 128 maycorrespond to (or be provided through) a set-top box. The integratedcontent 130 may be delivered as an EBIF formatted content.

In some variations, the distribution node 126 acquires the time basedmetadata for particular media content on a need to or real time basis.For example, in an implementation in which the playback device 128 isprovided as or through a set-top box, one more embodiments enable theset-top box or cable head end to request the time based metadata basedon user input. In such an implementation, the distribution node 126 maysupply the media content identifier to the service 120, in order toreceive the time based metadata 125 for the corresponding media content.

In another operational environment, the service 120 supplies the timebased metadata 143 to a playback device 138 that receives the mediacontent from another source. In particular, a media source 132 maycorrespond to, for example, a cable head end, or a website fordownloading content, or a media that contains stored media. In thelatter case, the media source 132 and media playback device 138 may beprovided from the same device of the user.

The user may operate the playback device 138 to render media content 135from the media source 132. According to some embodiments, the end usercan elect to enhance the media playback with the time based metadata 143provided through service 120. In one implementation, the user may make arequest 141 for time based metadata information from the service 120.The request 141 may specify the media content the user is about towatch. As an alternative or variation, the request 141 may be generatedfrom the media source 132, which responds to programmatic conditionsand/or user input.

Upon receiving the request, the service 120 obtains synchronizationtiming information 131 from the media source. In an embodiment in whichthe media source 132 corresponds to, for example, a cable head end orwebsite, the service 120 may query the media source 132 by specifyingthe media ID 133 for the media content that is to be rendered on theplayback device 138. The media source in turn may provide the service120 with the synchronization timing information 131. The synchronizationtiming information 131 may include information that precisely indicateswhen a content item is to initiate, or where (e.g. which scene or frame)in the timeline of a particular content is being rendered at the time ofthe query. As an alternative or variation, the service 120 may query,for example, the user's playback device 138, which can include mediacorresponding to the media source. For example, in some Blu-rayenvironments, the Blu-ray player may communicate with service 120 overnetwork in order to receive enhanced content in the form of time basedmeta-data 143. In some embodiments, the playback device may be atelevision.

In some implementations, the media source 132 may supply the timinginformation 131 as a one-time event for the particular media content,thus initiating a clock or counter on the service 120. Alternatively,the timing information 131 may be provided repeatedly, or on an ongoingbasis. With this information, the service 120 supplies time basedmetadata 143 to the playback device 138 in a manner that synchronizesthe timeline of the metadata 143 with that of the media content.

In another operational environment, the service 120 supplies enhancedcontent in the form of time based metadata and the associated mediacontent to a user end device for playback. The user and device maycorrespond to, for example, a tablet or personal computer that connectsto the service 120 (or its affiliates or partners) via, for example, awebsite or web service. In some variations, the playback device 148 maycorrespond to, for example, a Blu-ray player or television whichreceives content and metadata from over a network.

In the operational environment depicted, the user of the playback device148 makes a media request 147 from the service 120. The media request147 may identify an underlying media content, such as a movie orprogram. The service 120 supplies enhanced media 149 in that it includesboth media and its associated time based metadata. The service 120 maypre-synchronize the time based metadata with the underlying mediacontent so as to transmit the content and metadata concurrently or atone time. The playback device 148 may include programming resources thatenable it to render both the media content and the information orcontent item included with the metadata.

As a variation, the service 120 may be configured to supply the timebased metadata 157 to a paired device 160, in connection withcorresponding media being rendered or played back on a user playbackdevice 158. The user playback device 158 may correspond to a mediasource device, such as a Blu-ray player that plays back content from adisk, or tablet that plays back the content from the stored media file.Alternatively, the playback device 158 may receive the media 155 fromthe service 120. Still further, the playback device 158 may receive themedia 155 from any media source, such as a cable head end or website.

In the example shown, the media playback device 158 receives the mediafrom the service 120. Rather than communicating the time based metadata157 to the media playback device 158, the metadata 157 may becommunicated to the paired device 160. In one embodiment, the paireddevice 160 makes the request 161 for time based media data of anidentified media content. The paired device 160 may be configured toidentify itself as being a paired device, rather than a playback (orprimary playback) device. Alternatively, the service 120 may beconfigured to identify from the request 161 that the paired device 160is to receive the time based metadata 157 information for an identifiedmedia content, in which the media content is played back on a differentdevice or medium. In response to the request 161 the service 120supplies a time based metadata 157 to the paired device 160.

In an embodiment in which the service 120 provides the media 155, theservice controls transmission of both the media content and themetadata. As such, the service 120 is able to synchronize the deliveryof metadata information to the paired device 160, while transmittingmedia 155 to the playback device 158.

In an embodiment in which the media 155 is transmitted from analternative media source, such as a cable head end or website, theservice 120 may receive synchronization timing information for mediacontent being played back, so that the delivery of the time basedmetadata 157 to the paired device 160 is synchronized so that thetimelines of the metadata and the media content are aligned. Thesynchronization timing information may be received by querying the mediasource (e.g. cable head end), as similarly described with otherembodiments. Alternatively, the media playback device 158 may providethe necessary synchronization timing information, by initiating playbackof the media content upon the paired device 160 receiving the time basedmetadata 157.

In an embodiment in which the media 155 is stored or otherwise local tothe playback device 158, the paired device 160 may be operated torequest the time based metadata 161 that is associated with the mediacontent that is to be played back on the device 158. For example, theuser may operate the paired device 160 to identify the media contentthat is to be played back on the media playback device 158.Alternatively, the two devices 158, 160 may communicate with one anotheracross, for example, a local communication port (e.g. Bluetooth orWiFi). Still further, the playback device 158 may be triggered tocontact the service 120 in response to a playback request from a user(e.g. insertion or play command of a Blu-ray disc, play command forstored media file on a tablet etc.).

In response to receiving the request 161, service 120 may query themedia playback device 158 for synchronization timing information, inresponse to receiving the request 161. For example, a user may beassociated with an account on the service 120. Once the user accessesthe service 120, the service 120 locates and communicates with both ofthe user devices, thus enabling identification of the paired device forreceiving the time based metadata 157.

Still further, in some variations, the time based metadata 157 may becommunicated to the paired device 160 from the playback device 158. Forexample the playback device 158 may synchronize both the time basedmetadata and the media content by receiving respective transmissions ofone or both from an external source (e.g. content source or service120), and then retransmitting the metadata to the paired device 160. Thetransmission of the metadata 157 to the paired device 160 can be donethrough a communication medium such as, for example, Bluetooth or othershort radiofrequency wave communication channels.

FIG. 1B is a block diagram illustrating the architecture of the timebased metadata service(s) 120 according to some embodiments describedwith FIG. 1A. More specifically, According to some embodiments, theservices described in connection with the use of the time based metadataservice(s) 120 may include one or more components, modules or servicesshown in FIG. 1B depending on the operating environment of theimplementation.

With regard to embodiments described, time based metadata service(s) 120may also be configured to map multiple media playback devices to eachother thereby enabling the devices to share metadata and/or associatedcontent, even if one or more of the playback devices are not capable ofcommunicating directly with each other.

As described with FIG. 1A, in certain embodiments, the metadatageneration subsystem 110 processes media files 102 corresponding tomovies and/or programs. The output of the metadata generation subsystem110 includes a metadata file 111 having a combination of time stamps 113and associated information 115 or content (FIG. 1A).

In another embodiment such as illustrated in FIG. 1B, the metadata file111 may be sent from metadata generation subsystem 110 while theassociated information 115 or content may be provided from a secondsource, such as, for example, content service(s) 195. The time basemetadata service 120 can include a content service communicationinterface 173 to receive content (e.g. movies or programs) fromdifferent sources, either in real-time or asynchronously withre-transmission of such content in enhanced form. As examples, thecontent service communication interface 173 includes a program interfaceto receive content as file uploads from content providers. In othervariations, content interface 173 receives content in form of streamingor via broadcast medium (e.g. cable, satellite) or data communicationsonly (e.g. questions for a quiz, consumer user's commentary and webposts, lists of friends related to the content).

In some embodiments, time metadata service(s) 120 is configured toassociate or map the content (e.g. as received from content service(s)195) to the metadata file 111 and provide the newly associated contentto one or more playback devices. For example, the metadata file 111 maybe received via a metadata services(s) communication 170 of the timemetadata service(s) 120 and provided to a metadata content mappingcomponent 171. The mapping component 171 then associates or maps themetadata 111 with content (e.g., movie or programming content) receivedfrom the content service(s) 195 via content service(s) communication173. In some embodiments, the metadata 111 includes time stamps thatcorrelate to scenes, frames and/or events in the content. The metadatato content mapping 171 programmatically links the time stamps and theassociated metadata to the run time of the corresponding content. Forexample, one file, or two linked files may be created that can be playedsimultaneously by a playback device in order to render the content (e.g.movie) and time based metadata at appropriate times (e.g. frameindicated by the time stamp).

The service 120 communicates the mapped metadata/content to a playbackdevice using, for example, a delivery node 126 such as described abovewith respect to FIG. 1A. Other delivery nodes such as described withFIG. 1A may alternatively be used.

The service 120 may also include a user accounts management service 175.The service 120 may associate individual users with accounts, which canincorporate pay for service functionality or other capabilitiesrequiring the activation of an account. For example, functionalityprovided via the service 120 can be provided via a pay per view ormonthly subscription basis. As an addition or alternative, accountinformation can carry device and user profile information of aparticular user. Such information may include (i) contact informationabout the user; (ii) one or more identifiers as to a type or types ofplayback device(s) the user is utilizing; (iii) historical dataindicating the interactions of the user with a particular media file;(iv) linked accounts for other web services which the user may haveregistered to; and/or (v) preferences of the user regarding the outputand display of the supplemental content.

According to some embodiments, service 120 provides programmaticenhancements to enable third-parties (e.g. advertisers) to identifyappropriate topics and information to supply with content. The service120 includes a programmatic service communication (PSC) 176 thatprovides an interface for third-parties (e.g. advertisers, contentowners, movie studios or sponsors) to research content and time basedmetadata from a library or collection of media files (e.g. movie libraryof studio). The PSC 176 can be used by the third-party (e.g. via service190) in order to specify additional metadata content, such as, forexample, promotional metadata content. For example, the advertiser mayuse the PSC 176 to specify overlay ads or content that relate to aparticular commercial product, or, conversely, a particular commercialproduct approved for display with a particular media asset. Theadvertiser may also specify the media content, or supply information tohave the media content selected programmatically. The programmaticmapping component 177 can use information provided from the advertiserto perform one or more of the following: (i) select the media content onwhich the advertiser content is to be displayed, (ii) select thescene(s) or events of the media content, and/or (iii) identify theparticular time stamp within the media content that is to serve as thestamp for the supplied content.

As an addition or alternative, the programmatic mapping 177 may includelogic (algorithms) to scan media content, its associated metadata and/ortime base metadata in order to identify one or more semantic categoriescorresponding to the media being viewed by the user. This informationmay be communicated to the programmatic service(s) 190. The service canenable manual or programmatic selection of supplemental metadatacontent, in connection with time based metadata.

For example, if the user is watching a program or movie about dogsand/or one or more of the time stamps is identified as corresponding todogs, mapping 177 can communicate the information to the programmaticservice(s) 190. The service 190 may then determine which supplementalcontent (e.g., advertisements regarding dog food, dog breeders, etc.)can be added to the media and provided to the user.

In certain embodiments, service 120 may also include a deviceidentification service 178 configured to identify a type of playbackdevice utilized by a user. For example, the device identificationservice 178 may determine that a user has access to, or is using, aparticular device such as, for example, tablet computer, Blu-ray player,set-top box, etc., as well as the communication format for each device(such as determined by device communications 179). In one or moreembodiments, once the device information has been determined, theinformation may be associated with a user and stored using the useraccounts management 175. Device identification may also trigger theservice 120 to operate in one of multiple possible computingenvironments, as described with prior embodiments.

Service 120 may also include mapping tool 180 configured for associatingcontent received from programmatic mapping service(s) 190 to metadata111. An exemplary mapping tool 180 is shown in FIG. 9A.

Referring to FIG. 9A, mapping tool 900 includes a selector interface 910to enable a manual operator to filter the time stamps and/or metadatabased on scenes. The manual operator may utilize the interface through,for example, the service 190. For example, the interface 910 may begenerated or provided with the mapping tool 180. In using the interface910, the operator can view the context of numerous scenes in a mediacontent, view the time based metadata and its information, generate newtime based metadata, verify the additional information and/or selectother information, if any, that could be added. For example, if aparticular song by an artist is playing in the given scene, additionalcontent or advertisements corresponding to additional songs performed bythat artist may be selected by the operator to later be provided to theuser at the time of content playback. In another embodiment, songsand/or artists from a particular genre of the identified song and/orartist may be associated by the operator with the metadata which can belater presented to the user. The mapping tool 900 may be used on anynumber of categories such as, for example, clothing, food, scene orshooting locations, scene mood, atmosphere or ambiance, objects presentin screen, etc. As an additional example, the mapping tool may identifypresence of commercial objects, or objects that are identifiable to atheme or topic (e.g. advertisement theme), or other factors, includinguser specified or user generated (“crowd-sourced”) factors.

Referring back to FIG. 1B, service 120 may be configured to map oneplayback device to another playback device without each of the playbackdevices having capabilities of communicating directly with each other.Thus, a first playback device may deliver relevant information 115through service 120 to a second playback device or paired device. Incertain embodiments, device communications 179 determines acommunication protocol of one device and may convert the relevantinformation 115 from a first communication protocol utilized by thefirst playback device, to a second communication protocol, utilized bythe second playback device.

In one embodiment, the imported metadata and related time stamps may beimported in a first format (e.g., the format in which they were created)and transferred to a playback as is. In another embodiment, the timestamps and metadata are programmatically processed to match thestructure of the metadata file 111 and/or the playback device to whichthe information will be sent.

FIG. 1C-FIG. 1D illustrate variations or additions to embodimentsdescribed in FIG. 1A and/or FIG. 1B. FIG. 1C illustrates an architecturein which the service 120 provides the media 149 to a playback device(e.g., playback device 148). The playback device 158 communicates anaudio fingerprint 151 to the service 120. The audio fingerprint can beused to determine information about the media content, such as whichmedia content is being played back, or alternatively, which scene isbeing played back. The audio fingerprint 151 can be communicated to theservice and used to identify the media content, or the particularportion or scene of the media content. For example, the media ID 133 canbe determined from the audio fingerprint 151. This information isprovided to the service 120 which, under some embodiments, can map orassociate the relevant time stamps 113 to the identified media ID 133and provides the Information 115 back to the playback device 148.

In another embodiment illustrated by FIG. 1D, an audio signal 153associated with the media 149 may be played on a first device (e.g.,playback device 158) and listened to on a paired device (e.g., paireddevice 160) after which the paired device 160 provides the audiofingerprint 151 to the service 120. As discussed above, the audiofingerprint 151 is identified and mapped to a media ID 133 and theinformation is provided to the service 120. Service 120 then maps orassociates the relevant time stamps 113 to the identified media ID 133and provides the information 115 back to the paired device 160. Inanother embodiment, the relationship between the paired device 160 andplayback device 158 may be reversed such that the paired device 160transmits the audio signal to the playback device 158, which in turn,transmits the audio fingerprint to the service 120. In the variousimplementations shown, the paired device may be used to receivemetadata, either from the service or from the playback device 158, 160.The paired device can alternatively display portions of the content.

Methodology

FIG. 2 through FIG. 5 illustrate various embodiments for providing timebased metadata in connection with the rendering of a media content.Methods, such as described by FIG. 2 through FIG. 5, may be implementedusing components such as described with a system of FIG. 1. Accordingly,reference to elements of system 100 is intended to illustrate suitablecomponents or subsystems for performing a step or sub step beingdescribed.

FIG. 2 illustrates a method for generating time based metadata for amedia content, according to embodiments. The media content maycorrespond to, for example, a movie, a television program, a short film,an animation, a homemade video, or other audio/video work or creation.The media content may be processed or analyzed (210) using programmaticresources that facilitate manual operation and/or automation.Accordingly, embodiments may include a metadata generation sub-system110 that includes a manual editor interface (e.g. See FIG. 8), and oneor more programmatic resources that enable automated processing. Theautomated resources can include, for example, image analysis (214) whichuses image recognition to identify persons and objects appearing in thecontent. As an addition or alternative, the automated process caninclude steps for analyzing script and/or closed-captioned analysis(216), which uses text content associated with the content in order tofacilitate programmatic detection of events (e.g. the presence of aperson in a scene, the use of a location, etc.).

The media content is processed in order to identify the occurrence ofevents for which supplemental information or content can be provided(220). Various kinds of events may be detected in the media content.These include, for example, events pertaining to (i) an actor orcharacter (222), (ii) the use of objects (224), such as commercialobjects, (iii) the playing of songs or supplemental media (e.g.television show in background) as part of the media content, and/or (iv)depiction of locations (226) (e.g. landmarks, geographic regions etc.).

The events of the media content may be individually associated with timestamps that are included as part of an associated metadata file (230).The metadata file is structured to reflect a timeline of the mediacontent. The occurrence of an event in the media content may besignified by a trigger, which is marked in the metadata file to coincidewith the occurrence of the event at the playtime of the media content.In this way, individual triggers are associated with correspondingevents of the media content.

Once events are identified, the events may be linked to pertinentinformation or content (240). For example, time stamps that signify thepresence of an actor or character in a scene of the media content may belinked to a data item that includes additional information about theactor, the character or the scene. Each time stamp can be linked to adata file that contains additional information pertaining to the eventsignified by the time stamp. The data file may be createdprogrammatically and/or manually. For example, if an actor is signifiedby a time stamp, programmatic resources may automatically identify aprofile of the actor from a movie database and include information (or alink) to the movie profile as part of the data item that is associatedwith the time stamp. As an alternative or addition, an operator maymanually edit or create content for the data item that is associatedwith the trigger.

In some implementations, metadata generation subsystem 110 processes amedia content in order to detect events that occur as part of the mediacontent. The metadata generation sub-system 110 associates the detectedevents with time stamps, which are in turn associated with timinginformation that signifies when the events occur during the playback ofthe media content. Additionally, the detected events can be associatedwith information or content items that supplement the content duringportions that coincide with the placement of a corresponding time stamps(signifying when the event is taking place as part of the content). Inone implementation, the metadata generation sub-system 110 includes amanual editor interface (e.g. see FIG. 8) that includes programmaticenhancements for facilitating an operator in (i) inserting time stampsalong an appropriate timeline of the corresponding media content, (ii)associating time stamps with data items (e.g. information/content), orprogrammatic services 190 or content services 195 that pertain to (e.g.describe or supplement) the event identified by each of the time stamps.

As an alternative or addition, the process of event detection andsubsequent data item association may be performed programmatically andwith automation. In one implementation, individual frames that comprisethe video portion of the media content are analyzed using imageanalysis. For example, individual frames are analyzed to detect faces,and to associate the faces with identities of persons (e.g. actors) froma database. Likewise, frames of the media content may be analyzed todetect objects (e.g. commercial objects) and to recognize the objects bytype of brand using image analysis. The objects can then be mapped toprogrammatic services 190 or content services 195.

Once the metadata file is created with time stamps and linkedinformation, the metadata file is made available to manual operators. Incertain embodiments, the manual operators may utilize tools shown anddescribed with respect to FIG. 9A through FIG. 9C to check the accuracyof the time stamps, and the associations of those time stamps withrelevant information 115 and programmatic services 190 or contentservices 195. As an addition or alternative, such tools may be used bythird-party metadata providers.

The time stamps and metadata are then communicated to playback devicesand/or media sources for use with playback of the associated media file(250). Depending on the implementation, the contents of the metadatafile can be transmitted in real-time, to coincide with playback of theassociated media content on a playback device. Alternatively, thecontents of the metadata file can be transmitted and stored in aplayback environment, for use when the associated media content isplayed back from a particular source or on a paired device 160 at alater time. For example, in a cable/set-top box environment, themetadata file and/or its contents can be transmitted to a media source(e.g. cable head-end, website) where it is re-transmitted to a userplayback device (e.g. set-top box).

Alternatively, the metadata may be transmitted to the media source (e.g.cable broadcast, website). The receiving media source may combine amedia content transmission with the metadata, so as to transmit themedia content and the metadata to the user for playback. According tosome embodiments, the media source may include programmatic resourcesfor (i) requesting the metadata that is to accompany the media contenttransmission, and (ii) combining transmission of the metadata with themedia content, including synchronizing the delivery of metadata with thetransmission of the media content along a timeline that is reflected bythe media content.

Still further, the contents of the metadata file can be communicated tothe playback device while the playback device receives the media contentfrom another media source (e.g. cable/set top box, website). Forexample, the playback device may receive the metadata from time metadataservice 120, while receiving the coinciding media content from a mediasource (e.g. cable head end). The service 120 may synchronize thetransmission of the metadata using reference timing information for theplayback of the media content (e.g. start time). Alternatively, theplayback device may synchronize the receipt of metadata from one sourcewith the receipt of media content from another source in creating anenhanced playback containing both the media content and the time basedmetadata.

In some variations, the contents of the metadata file may be transmittedand stored by, for example, a user of a playback device. The stored filecan be accessed and used when the playback device renders the associatedmedia content or file.

As another variation or alternative, the metadata file, or its contents,may be transmitted to a paired device of a user, for use in conjunctionwith rendering of the associated media content on a playback device. Forexample, the metadata file may be transmitted over a network to a tabletdevice that the user can hold and view, while watching the associatedmedia content on a set-top box or Blu-ray environment.

Numerous variations to how the generated metadata is communicated andused are described with, for example, embodiments of FIG. 3 through FIG.5.

FIG. 3 illustrates a method for providing media content that is packagedor integrated with time based metadata, according to an embodiment. Asdescribed with an embodiment of FIG. 2, a metadata file is generated fora particular media content (310). The metadata file may be structured toinclude timing information that identifies the occurrence of events inthe runtime of the corresponding media content. The metadata file may beprovided as a separate file from the media content, but the packaging ofthe metadata with the media content can inherently include programmaticresources that can trigger the metadata file and the media content tosynchronously initiate their respective playback, so that the timelinefor each of the metadata and the media content are aligned during theplayback of the media content.

In an embodiment of FIG. 3, the metadata is packed with the media fileand then delivered (320). In one implementation, the combined media andmetadata are transmitted to a media source, such as a broadcast source(e.g. cable head end in distribution) or web site, where the data isretransmitted to a playback device of the user (322). In anotherimplementation, the combined media and metadata are transmitted to themedia playback device of a user without an intervening distribution ofbroadcast source (324). For example, service 120 may be provided througha website that can deliver the combined media and metadata package to anend user operating a computing device, such as a laptop, personalcomputer, game station, or Internet television. As another variation,the combined metadata and media may be provided on a storage medium,such as on a Blu-ray disc, or flash memory device (e.g. accessorydevice, tablet) (326).

FIG. 4 illustrates a method for providing time based metadata to a mediasource, for transmission to a media playback device with media content,under an embodiment. A method such as described with FIG. 4 may beimplemented to enable a media source to deliver or otherwise providemedia content in an enhanced form (as provided by time based metadata).In response to a media playback request to a user, a media source (e.g.broadcaster) may request metadata for the requested media content (410).In an alternative implementation, the request may be initiated by themedia playback device of the user, which may use a network connection tocommunicate with the service 120 separately from the media source. Theservice 120 may respond to the request by delivering metadata associatedwith the media content identified in the request to the media source(420). The media source may be provided with programmatic resources tosynchronize the timing of the metadata with the playback of the mediacontent (430). For example, the media source may include an applicationthat synchronizes the metadata with the playback of the media content,so as to transmit both metadata and media content in synchronous fashionto the media playback device.

In some variations, the media playback device may request and receivethe metadata independently from the media source. For example, the mediaplayback device may request the metadata from the service 120. Theservice 120 may acquire the synchronization timing information from themedia source, and deliver metadata in real-time to the playback device.Alternatively, the playback device may receive the metadata in real-timeor synchronously, and include programmatic resources for synchronizingthe playback of the metadata with the associated media content. Numeroussuch variations are possible.

FIG. 5 illustrates a method for providing time based metadata to aplayback device that receives the media content from a media source,according to another embodiment. In one implementation, a user requestsmetadata for a media content that the user is to playback (510). Themedia content may, for example, reside on the user's device, or beprovided from a separate media source (e.g. cable or website).

The service 120 may respond to the request in real-time, so as totransmit metadata concurrently and in-sync with playback of the mediacontent on the playback device. In responding to the request, service120 may acquire synchronization timing information in order to transmitthe media content in sync (520). The synchronization timing informationmay correspond to, for example, the exact time that the media contentwas initiated, or the precise frame or portion of the media content thatis rendered at a reference time. In one implementation, the service 120acquires the synchronization timing information from the media source(522). In another implementation, the service 120 acquires thesynchronization timing information from the media playback device (524).

In some variations, the media source may reside on a user device. Forexample, the media source may correspond to a Blu-ray player. Thesynchronization timing information may be communicated from the Blu-rayplayer of the user.

The service 120 may then transmit the metadata information in real-timeto the device of the user (530). In one embodiment, the metadata istransmitted to the media playback device of the user (532). In avariation, the metadata is transmitted to a paired device of the user(534). The paired device may correspond to, for example, a tablet, smartphone or laptop, while the media playback device may correspond to aset-top box and television, personal computer, Blu-ray player orInternet television. As an example, the service 120 may maintain anaccount for the user from which it maintains information to identify andlocate the media playback device and the paired device on a network. Inanother embodiment, the service 120 can broker a request. For example,service 120 may provide response actions to a user interacting with themedia, either through the playback device or paired device and connectthe user to programmatic services 190 or content services 195.

FIG. 6A through FIG. 6F illustrate examples of media contentpresentation that includes metadata content, according to someembodiments described herein. The enhanced presentation of media contentmay be implemented using, for example, a system such as described withan embodiment of FIG. 1. Accordingly, reference is made to elements ofFIG. 1 for purpose of illustrating suitable components or elements forcreating metadata presentations as shown.

FIG. 6A illustrates a media content presentation that is enhanced withpresentation of time based metadata, in accordance with one or moreembodiments. In an embodiment described, the media content is a movie,and the metadata that is presented with the movie includes actor namesfor characters that appear in individual scenes. In the exampleprovided, a given scene is marked by the presence of three actors. Eachare identified by metadata items 602. The metadata items 602 may alsoinclude additional information, such as the name of the character thatthe actor is playing in the given content item. Additionally, a song ortrack that starts playing in a scene may include an informational tag604. With reference to prior embodiments, the metadata file may beprovided in form of a file that includes time based triggers thatcoincide with the timeline of the movie. At scenes (e.g. frames) thatcoincide with introduction or presence of an event (e.g. an actor, asong playing), the informational tag is presented as an overlay.

FIG. 6B illustrates the information tag being interactive or able toinclude supplemental information, such as the filmography 606 of anindividual actor. In some implementations, the user is able to interactwith the information tags that appear over the media content. The userinteraction may, for example, be the user operating the playback orpaired device requesting additional or supplemental metadata from themetadata source (which may be the media source or the service 120). Inanother embodiment, the user interaction may be a request to receive(e.g., video on demand, purchase a digital copy, or Blu-ray copy etc.)and/or view additional media content the selected actor(s) is in.

FIG. 6C illustrates additional examples of metadata content that can beoverlaid with the media content. In FIG. 6C, the metadata content 612can occupy a majority of the screen space, so as to present additionalinformation about the actor (or event that is being described).

FIG. 6D illustrates an example in which metadata includes supplementalinformation or services, such as an interactive feature that enables theuser to purchase a product associated with an event of media content (orthe media content itself). In the example shown, the user is able topurchase a soundtrack by interacting with metadata content 622. Themetadata content 622 can be displayed to the user at, for example, theoccurrence of a relevant event, such as the playback of a song thatappears on the soundtrack.

FIG. 6E and FIG. 6F illustrate the user of metadata content thatcoincide with product placement within a media content. In the exampleshown, the product placement event coincides with an actor of atelevision show riding in a sports car. The metadata content 624identifies, for example, the make and model of the car. Additionalinformation, such as information about the character 626 or program, mayalso be displayed as an overlay of the content.

FIG. 7A through FIG. 7F illustrate examples of media contentpresentations that include metadata content for playback and/or paireddevices, according to some embodiments described herein. In FIG. 7A andFIG. 7B, a user is able to login to a service that is provided inconnection with the playback of the media content. For example, the usermay include an account with service 120, and have the ability to loginin order to receive metadata in connection with media content that iseither received from the service, from a different media source (e.g.cable or website), or played back from a local source (e.g. Blu-rayplayer). FIG. 7C illustrates an implementation in which the metadatapresentation over the media content is personalized. In variations, themetadata content presentation may also be configured or tailored for theparticular user based on, for example, preferences of the user. Forexample, as shown in FIG. 7C, the user preference may be to have allspoken dialogue displayed in real time.

FIG. 7D through FIG. 7F illustrate presentations that supplement themedia content independently of the media content presentations. Suchsupplemental presentations may be displayed on a playback device in lieuof, for example, the media content (e.g. the media content may bepaused), or displayed on a paired device (e.g. tablet) while the userwatches the media content on a media playback device (television).Accordingly, some implementations provide for the supplemental metadatacontent to be provided independent of the occurrence of events ortriggers. For example, the supplemental content showing the websitecontent may be accessible to the user throughout the playback of themedia content.

In FIG. 7D and FIG. 7E, the presentations include, for example, websitecontent which can display various forms of content about individualcharacters of a program. In FIG. 7F, the supplemental content includesonline auction or e-commerce content from programmatic services 190 orcontent services 195. Various alternatives to the form of supplementalcontent that can be displayed are possible.

Metadata Placement and Development

Various parties may utilize an interface or tool to develop metadatathemes of content for inclusion with a media content. For example, anoperator of the metadata generation sub-system 110 may place metadataaccording to a default criteria (e.g. identify all actors, songs,bibliography etc.). Other operators, such as users of the service 190may place metadata by themes. For example, the operators may be providedan interface (e.g. See FIG. 9A and FIG. 9C) where metadata isidentified, and commercial or user-selected themes can be specified forscene(s). The operator can view the content, the product placement, seetags associated with placed content etc. to develop product placement,themes and other content.

For example, referring to FIG. 9C, a user may be able to check therelevant information in a particular scene such as items of clothingworn by the actors, information regarding the character played by theactor, and/or a story line associated with the scene. Additionally, theuser interface tool of FIG. 9C may present metadata information 115 asan overlay to the media file 102. As such, a manual operator may verifythe accuracy of the metadata mapping to the time stamps 113.Additionally, the manual operator may also determine the relevance ofthe information to the content of the media. In such embodiments, theuser interface tool is dynamic, and the visual elements of the interfaceappear and disappear as the timeline of the media 115 is traversed.

Referring to FIG. 9C, a web based tool may be provided to enable themanual operator to selectively verify information in a particular scene.For example, the manual operator may be able to select one or morecategories of the supplemental content to verify (e.g., products,portrayals etc.). As a result, only information corresponding to theselected categories will be displayed. Additionally, the manual operatormay be able to add events and/or content using this tool.

Second Screen Examples

FIG. 10A-10B illustrate examples of media content interfaces generatedusing metadata content for a paired device, according to embodimentsdescribed herein. As discussed above regarding FIG. 1, a paired device160 may receive time based metadata, such as from metadata service 120,in connection with corresponding media being rendered or played back ona user playback device 158. Embodiments provide for the programmaticservices on the paired device 1000 (e.g. a phone, tablet or personalcomputer) receiving metadata corresponding to a media being rendered orplayed back on a main playback device, and providing selectable featuresto enable a user to access supplemental content related to the receivedmetadata in an interface 1000. The interfaces of FIGS. 10A and 10B maybe in lieu of or in addition to any interfaces displayed on the mainplayback device. An indicator 1002 may be provided to specify to a userthat the features being displayed include content relating to the mediaplayback, relative to the beginning and end of the media.

The interfaces of FIG. 10A-10B include a plurality of selectablefeatures 1004 that relate to persons or objects appearing in the media.The features include graphical and/or text content that describe personsor objects appearing in the media being rendered or played back on theuser playback device, and further describe supplemental content whichmay be generated and displayed in response to a user selection of thefeature. For example, the graphical content of the feature may includean image from the media being rendered or played back on the userplayback device, and when selected may cause the interface to be removedand a supplemental content (e.g. graphic and/or text content) relatingto the media and/or feature to be displayed on the paired device. Forexample, generating the supplemental content may include, in response toa user selection of a selectable feature, causing a larger version ofthe image and a text caption to be displayed on the paired device.

In the embodiment of FIG. 10A, a series of features 1004, 1006 and 1008are displayed on the interface 1000. Embodiments provide for thefeatures to be selectable by a user. For example, where the paireddevice is a mobile computing device that includes a touch screendisplay, and the interface 1000 is provided on the touch screen display,in an embodiment the user may select a particular feature by touchingthe portion of the screen corresponding to the selectable feature. Insuch embodiments, service 120 can detect selection of the feature andprovide supplemental content on the main playback device and/or on thepaired device 1000 corresponding to the selected feature. Thesupplemental content may include, for example, information about actorsor objects that relate to a description of the selected feature, and theinformation may be displayed on the paired device. For example, inresponse to selection of feature 1006, the service 120 may cause contentto be displayed on the paired device relating to characters who appearin the scene and/or the graphic of the feature.

FIG. 10B illustrates a media content interface in an embodiment of theinvention. A series of selectable features 1010 are displayed on theinterface. When selected, the selectable features 1010 may causesupplemental content to be displayed on the user playback device and/orthe paired device. The example of solid-square features 1012, whichdisplay solid squares without any graphical or text content, may bereplaced with selectable features 1010 by programmatic services on thepaired device including graphical and/or text content based on timebased metadata such as supplied by metadata services 120. Similarly, theselectable features 1010 may be replaced by solid-squares features 1012(or other selectable features) based on the time based metadata.

Link Sharing

FIG. 11 illustrates a method for providing media data to be shared orpublished, according to an embodiment. A method such as described by anembodiment of FIG. 11 can be implemented using components such asdescribed with other embodiments (e.g., a system of FIG. 1).Accordingly, reference may be made to elements of FIG. 1 for purpose ofillustrating suitable components for performing a step or sub-step beingdescribed.

With reference to FIG. 11, a user identifies a movie segment to share(1110). Referring to FIG. 1, for example, a user can specify input thatcorresponds to the movie segment using user playback device 158 and/oron the paired device 160 (1112). The input can identify the start/finishtimes, in reference to a movie's runtime. In this way, the timingspecification can designate a beginning time and an end time of themovie segment.

In some variations, the user input can identify a predefined segment(1114). The predefined segment may have a set of predefined features,for example a predefined start/finish time.

The user playback device 158 and/or the paired device 160 may respond tothe user input, by, for example, communicating the identified segment toa service or set top box, etc. (1116). The paired device 1160 can, forexample, determine, from user input, the start/end times for a movie,and then communicate the start/end times (along with an identifier ofthe movie) to the service or set top box.

In some variations, for example, a user can operate the paired device160 in order to provide input that identifies the desired segment(1118). For example, a user's interaction with a second screen (e.g.,touch screen) can be used to identify a movie segment or its startand/or end times. In such an embodiment a second screen interaction 1118may be provided to a user in response to determine, for example, apredefined segment. Various other input mechanisms and/or interfaces canbe used to enable a user to specify an input corresponding to a segmentof the movie.

After responding to a user input and/or second screen interaction, alink is generated (1120). The link may be generated to include, forexample, data elements that provide or correspond to one or more of thefollowing: (i) data that identifies a location of source file for themovie or movie segment (1122), (ii) start and end times for the segment(1124), (iii) time-based metadata that is associated with the segment ofthe movie (persons and/or objects that appear or are referenced in theidentified segment) (1126). In some variations, an identifier for themovie segment can be identified by the link. As an example, the link canbe a selectable hypertext link that, when triggered, causes theidentified movie segment 1110 to be played or otherwise displayed. Thegenerated link can be published (1130). This can include publishing thelink on a social networking account (1132) (e.g. Facebook, or any socialnetwork known in the art) or sharing the link through messaging (1134).For example, the link can be shared by sending the link by e-mail, textmessage, or another direct form of communication.

FIG. 12A-12E illustrate example interfaces for providing media data tobe shared or published, according to an embodiment. In the exampleprovided, the interfaces can be provided for use on, for example, amobile computing device or tablet. For example, in FIG. 12A, theinterface 1210 can enable a user to provide inputs through interactionwith, for example, a mobile computing device 1200. In some examples, themobile computing device (or tablet or other computing device) canoperate as a second screen.

In the example provided, the user triggers a selectable feature 1210.The feature 1210 can programmed to, for example, identify a segment, oralternatively provide user input that identifies a start/finish for themovie segment. In the example provided, once the segment is identified,time-based metadata (e.g., related shopping panel 1214) can be displayedto the user. The user input can also be used to enable the user topublish the clip 1215 of the segment. For example, the user cancommunicate the clip 1215 to the service, either with or without, forexample, the time-based or rich metadata.

FIG. 12B illustrates an interface where a user initiates sharing orpublishing of the clip 1215 by preparing an email message. The user maydesignate a recipient of the email message. A subject line canoptionally be automatically pre-populated based on, for example,metadata associated with the selected segment or movie.

FIG. 12C illustrates an interface where the user may view an image forinclusion in the email message. The image may, for example, correspondto a frame from the clip.

In FIG. 12D, the interface can enable the user to provide commentary,such as a text comment, with the link that is shared. For example, thetext commentary can be provided with an email message. For example, auser may provide a comment (“This is my favorite moment in this show”)which will be included in the email message. In variations, theinterface can enable the user to provide the commentary in connectionwith a link that is published on, for example, a social networkingaccount.

FIG. 12E illustrates an interface where the user has shared the clip bysending the email message. The message included an image correspondingto the clip, a text comment prepared by the user, and the publishedlink.

Computer System

FIG. 13 is a block diagram that illustrates a computer system upon whichembodiments described herein may be implemented. For example, in thecontext of FIG. 1A, system 100 may be implemented using a computersystem such as described by FIG. 13. In particular, service 120 (orportions thereof) may be implemented on a computer system such as shownand described with FIG. 13. The computer service 120 may communicatewith playback devices, paired devices and/or distribution sources ofmedia content and/or associated metadata using a network such as theInternet.

Additionally, a metadata generation sub-system 110 may be implementedusing a computing system such as shown and described by FIG. 13.

In an embodiment, computer system 1300 includes processor 1304, mainmemory 1306, ROM 1308, storage device 1310, and communication interface1316. Computer system 1300 includes at least one processor 1304 forprocessing information. Computer system 1300 also includes a main memory1306, such as a random access memory (RAM) or other dynamic storagedevice, for storing information and instructions to be executed byprocessor 1304. Main memory 1306 also may be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 1304. Computer system 1300 mayalso include a read only memory (ROM) 1308 or other static storagedevice for storing static information and instructions for processor1304. A storage device 1310, such as a magnetic disk or optical disk, isprovided for storing information and instructions. The communicationinterface 1316 may enable the computer system 1300 to communicate withone or more networks through use of the network link 1320.

Computer system 1300 can include display 1312, such as a cathode raytube (CRT), a LCD monitor, and a television set, for displayinginformation to a user. An input device 1314, including alphanumeric andother keys, is coupled to computer system 1300 for communicatinginformation and command selections to processor 1304. Othernon-limiting, illustrative examples of input device 1314 include amouse, a trackball, or cursor direction keys for communicating directioninformation and command selections to processor 1304 and for controllingcursor movement on display 1312. While only one input device 1314 isdepicted in FIG. 13, embodiments may include any number of input devices1314 coupled to computer system 1300.

Embodiments described herein are related to the use of computer system1300 for implementing the techniques described herein. According to oneembodiment, those techniques are performed by computer system 1300 inresponse to processor 1304 executing one or more sequences of one ormore instructions contained in main memory 1306. Such instructions maybe read into main memory 1306 from another machine-readable medium, suchas storage device 1310. Execution of the sequences of instructionscontained in main memory 1306 causes processor 1304 to perform theprocess steps described herein. In alternative embodiments, hard-wiredcircuitry may be used in place of or in combination with softwareinstructions to implement embodiments described herein. Thus,embodiments described are not limited to any specific combination ofhardware circuitry and software.

Although illustrative embodiments have been described in detail hereinwith reference to the accompanying drawings, variations to specificembodiments and details are encompassed by this disclosure. It isintended that the scope of embodiments described herein be defined byclaims and their equivalents. Furthermore, it is contemplated that aparticular feature described, either individually or as part of anembodiment, can be combined with other individually described features,or parts of other embodiments. Thus, absence of describing combinationsshould not preclude the inventor(s) from claiming rights to suchcombinations.

1. (canceled)
 2. A method comprising: receiving an indication of outputof media content; and sending, to an output device, based on theindication of output of the media content, a plurality of pieces oftime-based metadata, wherein each piece of time based metadata comprisesa timestamp corresponding to a point in time along a timeline of themedia content and indicates supplemental content associated with anevent in the timeline of the media content at the point in time.
 3. Themethod of claim 2, wherein each piece of time-based metadata facilitatesaccess to the supplemental content.
 4. The method of claim 2, whereineach piece of time-based metadata causes the output device to output thesupplemental content at the point in time.
 5. The method of claim 2,wherein each piece of time-based metadata causes the output device toretrieve the supplemental content from a server.
 6. The method of claim2, wherein the supplemental content is output by the output device basedon a detection by the output device of the timestamp.
 7. The method ofclaim 2, wherein the supplemental content comprises one or more oflocation information associated with the event, audio informationassociated with the event, or textual information associated with theevent.
 8. The method of claim 2, wherein the event comprises one or moreof an appearance of a character in the media content, an appearance of aproduct in the media content, or an indication of a particular locationin the media content.
 9. The method of claim 2, wherein the supplementalcontent is output as an overlay during the output of the media content.10. The method of claim 2, wherein sending the plurality of pieces oftime-based metadata comprises sending a package comprising the mediacontent and the plurality of pieces of time-based metadata.
 11. A methodcomprising: receiving, by an output device from a computing device, aplurality of pieces of time-based metadata, wherein each piece of timebased metadata comprises a timestamp corresponding to a point in timealong a timeline of media content and indicates supplemental contentassociated with an event in the timeline of the media content at thepoint in time; and causing, based on the time-based metadata and at thepoint in time, output of the media content and the supplemental content.12. The method of claim 11, wherein each piece of time-based metadatafacilitates access to the supplemental content.
 13. The method of claim11, wherein each piece of time-based metadata causes the output deviceto output the supplemental content at the point in time.
 14. The methodof claim 11, wherein each piece of time-based metadata causes the outputdevice to retrieve the supplemental content from a server.
 15. Themethod of claim 11, wherein the supplemental content is output by theoutput device based on a detection by the output device of thetimestamp.
 16. The method of claim 11, wherein the supplemental contentcomprises one or more of location information associated with the event,audio information associated with the event, or textual informationassociated with the event.
 17. The method of claim 11, wherein the eventcomprises one or more of an appearance of a character in the mediacontent, an appearance of a product in the media content, or anindication of a particular location in the media content.
 18. The methodof claim 11, wherein receiving the plurality of pieces of time-basedmetadata comprises receiving a package comprising the media content andthe plurality of pieces of time-based metadata.
 19. The method of claim11, wherein the supplemental content is output as an overlay during theoutput of the media content.
 20. An apparatus comprising: one or moreprocessors; and memory storing instructions which, when executed by theprocessor, cause the apparatus to: receive an indication of output ofmedia content; and send, to an output device, based on the indication ofoutput of the media content, a plurality of pieces of time-basedmetadata, wherein each piece of time based metadata comprises atimestamp corresponding to a point in time along a timeline of the mediacontent and indicates supplemental content associated with an event inthe timeline of the media content at the point in time.
 21. Theapparatus of claim 20, wherein each piece of time-based metadatafacilitates access to the supplemental content.
 22. The apparatus ofclaim 20, wherein each piece of time-based metadata causes the outputdevice to output the supplemental content at the point in time.
 23. Theapparatus of claim 20, wherein each piece of time-based metadata causesthe output device to retrieve the supplemental content from a server.24. The apparatus of claim 20, wherein the supplemental content isoutput by the output device based on a detection by the output device ofthe timestamp.
 25. The apparatus of claim 20, wherein the supplementalcontent comprises one or more of location information associated withthe event, audio information associated with the event, or textualinformation associated with the event.
 26. The apparatus of claim 20,wherein the event comprises one or more of an appearance of a characterin the media content, an appearance of a product in the media content,or an indication of a particular location in the media content.
 27. Theapparatus of claim 20, wherein sending the plurality of pieces oftime-based metadata comprises sending a package comprising the mediacontent and the plurality of pieces of time-based metadata.
 28. Theapparatus of claim 20, wherein the supplemental content is output as anoverlay during the output of the media content.
 29. An apparatuscomprising: one or more processors; and memory storing instructionswhich, when executed by the processor, cause the apparatus to: receive,from a computing device, a plurality of pieces of time-based metadata,wherein each piece of time based metadata comprises a timestampcorresponding to a point in time along a timeline of media content andindicates supplemental content associated with an event in the timelineof the media content at the point in time; and cause, based on thetime-based metadata and at the point in time, output of the mediacontent and the supplemental content.
 30. The apparatus of claim 29,wherein each piece of time-based metadata facilitates access to thesupplemental content.
 31. The apparatus of claim 29, wherein each pieceof time-based metadata causes the apparatus to output the supplementalcontent at the point in time.
 32. The apparatus of claim 29, whereineach piece of time-based metadata causes the apparatus to retrieve thesupplemental content from a server.
 33. The apparatus of claim 29,wherein the supplemental content is output by the apparatus based on adetection by the apparatus of the timestamp.
 34. The apparatus of claim29, wherein the supplemental content comprises one or more of locationinformation associated with the event, audio information associated withthe event, or textual information associated with the event.
 35. Theapparatus of claim 29, wherein the event comprises one or more of anappearance of a character in the media content, an appearance of aproduct in the media content, or an indication of a particular locationin the media content.
 36. The apparatus of claim 29, wherein receivingthe plurality of pieces of time-based metadata comprises receiving apackage comprising the media content and the plurality of pieces oftime-based metadata.
 37. The apparatus of claim 29, wherein thesupplemental content is output as an overlay during the output of themedia content.